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Abstract 

A  straightforward  algorithm  for  the  multiple  time  scale  decomposition  of  singularly  perturbed 
Markov  processes  has  been  presented  j*-fl72ff^That  algorithm  provides  a  uniform  approximation  of 
the  probability  transition  function  over  the  interval  t  >  0  through  the  construction  of  a  sequence  of 
a6Sre6ate  models  valid  at  successively  slower  time  scales.  When  only  the  structure  of  these  models 
is  desired,  the  algorithm  can  be  expressed  simply  in  terms  of  graphs  associated  with  each  of  the 
a6Sre8ated  models.  The  major  computation  then  becomes  computing  shortest  paths  in  these  graphs. 
This  representation  of  the  algorithm  furthermore  allows  analysis  of  more  complex  systems  where  there 
are  multiple  perturbation  parameters  with  unknown  relative  orders  of  magnitude. 


Keywords:  Markov  processes;  singular  perturbation,'  multiple  time  scales,' graph  theory. 


1  Introduction 

In  this  paper,  an  algorithm  for  the  analysis  of  the  multiple  time  scale  structure  of  a  perturbed  Markov 
process  is  developed  which  builds  directly  on  the  multiple  time  scale  decoposition  algorithm  presented  in 
[1,2].  Structure  of  a  perturbed  Markov  process  refers  to  the  complete  multiple  time  scale  decomposition 
of  the  type  performed  in  [1,2, 3, 4]  where  the  evolution  of  the  state  probabilities  is  approximated  using  a 
sequence  of  aggregated  models,  each  valid  a  successively  slower  time  scales.  In  this  case,  however,  only 
the  position  of  the  nonzero  transition  rates  of  each  of  the  unperturbed,  aggregated  time  scale  models,  and 
the  sets  of  states  which  constitute  the  aggregate  classes,  are  determined.  Although  the  detailed  behavior 
of  the  system  cannot  be  recovered  from  these  descriptions,  much  useful  information  is  retained.  It  will 
be  shown  that  this  subset  of  information  about  the  decomposition  can  be  obtained  using  a  very  simple 
graph-theoretic  algorithm  which  is  implicit  in  the  algorithm  presented  in  [1,2]. 

Use  of  these  structural  aspects  of  a  Markov  chain  can  have  many  applications.  For  instance,  an  algo¬ 
rithm  similar  to  that  presented  in  this  paper  has  been  applied  to  the  analysis  of  the  behavior  of  “simulated 
annealing”  optimization  methods  [5].  Other  applications  include  computing  order  of  magnitudes  for  the 
time  of  events  in  systems  with  rare  transitions.  Specific  applications  include  analysis  of  the  failure  time 
of  a  fault-tolerant  system  or  the  error  rate  of  a  communication  protocol. 

The  nature  of  the  decomposition  algorithms  presented  in  this  paper  are  very  different  from  those  which 
attempt  to  approximate  the  detailed  behavior  of  system.  In  those  algorithms,  numerical  calculations  of 
dominant  eigenvectors  of  unperturbed  systems,  and  of  <-dependent  “trapping”  probabilities  were  required. 
The  algorithms  in  this  paper  basically  consist  of  computing  various  types  of  connectivity  in  labeled  graphs. 
Since  the  computations  involve  only  integer  quantities,  issues  of  numerical  stability  are  not  relevant.  Also, 
by  introducing  the  graph-theoretic  formalism,  it  is  possible  to  employ  standard  graphical  algorithms  for 
computing  quantities  such  as  shortest  paths  between  vertices  of  a  graph. 

This  paper  is  organized  as  follows.  In  the  next  section,  the  decomposition  algorithm  and  uniform 
approximation  result  present  in  [1,2]  is  stated  and  a  simple  example  is  provided.  In  the  next  section, 
the  graphical  formalism  is  introduced  and  the  new  graphical  decomposition  algorithm  is  presented.  The 
final  section  includes  conclusions  and  a  discussion  of  the  relationship  of  this  approach  to  other  work. 
Also,  the  application  of  this  algorithm  to  models  with  multiple  perturbation  parameters  is  discussed  with 
application  to  a  fault-tolerant  system  model. 

*Thi«  research  *u  conducted  under  the  lupport  of  the  Air  Force  Office  of  Scientific  Research  under  grant  AFOSR-82-0258 
and  the  Army  Research  Office  under  grant  DAAG-29-84-K005. 
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2  Markov  Process  Decomposition  Algorithm 

In  this  section,  the  basic  algorithm  presented  in  [1,2]  is  stated.  The  reader  is  referred  to  those  works 
for  a  detailed  interpretation  of  this  algorithm,  proof  of  the  uniform  convergence  of  the  approximation, 
and  demonstration  of  its  relationship  to  the  algorithms  of  Courtois  [4]  and  Coderch  [3].  Following  the 
algorithm,  a  simple  exampie  is  provided. 

Algorithm  1  (Continuous  Time)  Begin  with  the  generator1  of  a  finite-state  Markov  process 

with  one  ergodic  class  for  e  >  0.  aj°'(f)  is  the  probability  transition  rate  from  state  i  to  state  j.  Set  k  *—  0. 

1 .  Partition  the  state  set  into  the  ergodic  classes  E i,  Ej, . . . ,  Eft  and  the  transient  set  T  generated  by 
AW  =  A(*)(0).  If  there  is  only  a  single  class  (N  —  \),  the  iteration  is  terminated. 

2.  For  each  class  Ej,  compute  the  ergodic  probabilities  Vt  6  Ei  of  the  member  slates  corresponding 

to  the  generator  Also  set  uj*'  =  0,  Vj  £  Ej. 

3.  For  each  state  j  and  each  class  Ej,  compute  terms  Cj*'(e)  such  that 

=  .;;’(<)(!+ oh)  tu 

E  «!?,’(*)  =  i  (2) 


K  =  1 


where 


(0 


=  Pr^*)(«,f)6^/  |  T7(*)(e,0)  =  i>  f  =  inf(t  |  .,(*>(«,  i)  *  T)^j 


(3) 


and  T^k\e,  i)  is  a  sample  path  of  the  Markov  process  generated  by 
Form  the  n  x  N  and  N  x  n  matrices 


Then 


Set  k  <—  k  +  1.  Go  to  1. 


UW  =  [«<$>]  and  Vlk\c)  =  [r^CO] 


A(*+i)(e)  _  iv-(*)(e)^(*)(e)CT(*) 


(4) 

(5) 


The  overall  approximation  of  the  evolution  of  the  transition  probabilities  4>^P(e,t)  (from  state  i  to  state 
j)  can  be  written  in  matrix  form  as 

^J7(°) *( 1  > (ei) V(°)  -  UWV^  + 

^(7(o)y(i)$(2)(tJt)VC1)v^°5  -  [/(°)r;(1)V'(l)v(0>)  + 


•  •l/(*-1)4>(*l(e*t)F(*“1). . .  y(°)  - 

1/1°)- .  . .  y(°)^  +  o(e) 


(6) 


or 


where 


*<*>(«,  r)  =  *<‘)(T)  =  eA(k)r,  and  =  V<k>(0) 

and  where  O(e)  is  a  function  of  e  and  t  which  converges  uniformly  to  zero  over  t  >  0. 


(7):a; 

□ 


A  simple  application  of  this  algorithm  is  presented  here.  The  same  system  will  be  analyzed  using  the  '  _ ^ 

graphical  decomposition  algorithm  presented  in  the  next  section.  \ty  Codn| 
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Figure  1:  Perturbed  Markov  process 


Example  1  Consider  the  generator  of  a  Markov  process 

Al°\c)  = 


-e  0  10 

0  -e  1  0 

e  £  -2  -  £  0 

0  0  f  0 


associated  with  the  state  transition  diagram  in  Figure  1.  The  ergodic  classes  and  transient  set  are 

Ei  =  {1},  Et  =  {2},  E3  =  {4},  T  =  {3} 

The  ergodic  probabilities  are  all  degenerate  in  this  case,  therefore 


U[°)  =  U<$  =  UW  =  1  or  l/W  = 


Suitable  terms  v(°)(e)  which  satisfy  (l)-(2)  above  are 

^(e)  =  *&{*)  =  1/2  -  e/4  w  ^(0)/e)  _ 

»33 '(<)  =  «/' 2 


1  0  0 
0  1  0 
0  0  0 
0  0  1 


1  0  1/2  —  £/4  0 
0  1  1/2  —  £/4  0 
00  £/2  1 


(8) 


0) 


(10) 


(11) 


Using  these  terms,  A(l)(e)  computed  using  (5)  generates  the  process  illustrated  in  Figure  2  and  given 


A*1^)  = 


-1/2 -£/2  1/2  0 

1/2  -1/2  —  f/2  0 

e/2  e/2  0 


(12) 


The  procedure  is  iterated  to  produce  A^^e)  and  A(2)(f).  A(2)(0)  generates  a  single  ergodic  class  and 
therefore  the  iteration  is  terminated.  The  set  of  £ -independent  Markov  models  from  which  the  approxima¬ 
tion  is  derived  is  shown  in  Figure  3.  □ 


jii  jiji  ji  uj  j 
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Figure  2:  Order  l/«  time  scale  model  for  Example  1 


3  Graphical  Decomposition 

The  uniform  approximation  (6)  produced  using  Algorithm  1  is  expressed  as  a  combination  of  the  behavior 
of  a  set  of  c-independent,  aggregated  Markov  processes  with  generators  Al°l(0), . . . ,  A(*)(0)  and  some 
associated  matrices  (the  £f(*>  and  which  characterize  the  recurrent  and  transient  states  of  these 

processes.  The  algorithm  can  be  greatly  simplified  if  only  the  constituent  states  of  the  aggregate  classes 
and  the  allowable  (i.e.  nonzero  rate)  state  transitions  are  desired.  In  this  section,  the  structure  of  the 
graphical  representation  is  presented.  Then,  the  graphical  algorithm  for  the  derivation  of  this  entire 
structural  decomposition  is  described. 

Given  a  perturbed  Markov  generator  A^(f),  an  associated  graph  G ^  =  (Jf(k\£(k))  can  be  con¬ 
structed  as  follows.  If  generates  an  n  state  process,  then  the  set  of  vertices  is  ~  {1,2, . . .,  n.}. 

Each  edge  is  a  triple,  e  =  (i,j,w)  £  £(*1  of  the  initial  vertex,  i,  the  final  vertex,  j,  and  a  nonnegative 
integer  weight,  w.  For  each  nonzero  entry  aj^(e),  j  £  »,  a  directed,  weighted  link  from  vertex  i  to  vertex 
j  is  introduced.  If  aj**(«)  is  strictly  0(c"),  the  weight  of  the  edge  is  w.  It  is  also  useful  to  construct  the 
“zero  weight”  graph  G ^  =  (A/W,£^)  fr°m  the  generator  Atk>(0).  The  set  of  vertices  is  unchanged, 
and  the  new  set  of  edges,  £^  C  £(k\  corresponds  to  the  subset  of  zero-weight  edges  in  £(*). 

Several  properties  of  the  graph  G  =  will  be  used.  First,  define  the  distance  between  two 

vertices  d(i,  j)  as  the  minimum  sum  of  weights  along  directed  paths  from  i  to  j.  If  there  is  no  such  path, 
then  d(i,  j)  =  oo.  A  communicating  class  C  is  a  maximal  set  of  states  such  that  t,  j  £  C  if  and  only  if 
d(i,  j)  <  oo.  A  class  C  is  final  if  there  is  no  vertex  j  £  C  such  that  d(i,  j )  <  oo  for  any  i£  C.  Also  define 
the  distance  ds(i,  C)  as  the  minimum  weight  path  from  i  to  the  class  C  with  no  intermediate  vertices 
in  the  set  E.  There  is  a  direct  association  between  the  Markov  process  generated  by  A  and  the  graph 
G.  Most  important,  an  ergodic  class  generated  by  A  exactly  corresponds  to  a  final  class  of  G.  Also,  a 
transient  state  of  A  corresponds  to  a  vertex  of  G  in  some  non-final  class. 

The  basic  structural  decomposition  algorithm  can  be  specified  in  the  graphical  terms  described  above. 
We  begin  with  a  graph  G 1°)  constructed  using  the  Markov  generator  At°^(f).  Then  a  sequence  of  graphs 
associated  with  successive  time  scales,  G q°\  G  . . . ,  GqK ^  is  constructed  such  that  Gq*  is  associated  with 
the  generator  A^(0)  constructed  using  Algorithm  1.  Also,  the  constituent  states  in  an  aggregate  class 
associated  with  each  of  the  vertices  in  the  graph  are  computed.  The  set  l/jk^  is  the  set  of  states  of  t ) 
associated  with  the  aggregate  class  represented  by  vertex  t  in  the  graph  G The  set  vjk^  is  the  set  of 
states  which  have  an  0(1)  probability  of  entering  the  aggregate  class  lljk *  by  the  kth  time  scale. 

Algorithm  2  (Structural  Decomposition)  Begin  with  the  graph  G^  =  constructed  from 

the  Markov  generator  A^°^(e)  of  the  n  state  process  T)W(e,t). 

Define  =  {»},  V,-0^  =  {t},  i  =  1, 2, . . .,  n. 

Set  k  «—  0. 

1.  Construct  the  graph  G§^  =  (A/^k\£o*^)  using  the  zero-weight  subset  of  edges  £q4'  C  £(k).  Identify 
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the  final  communicating  classes  of  G§\  E*jk\  I  =  1,2 The  complementary  set 

r(*)  =  AT(*) -£■(*)  ,  £■<*>  =  (J  £t5*)  (13) 

I 

is  the  set  of  vertices  in  non-final  classes.  If  there  is  only  one  final  class  (N  =  1  j,  then  the  decom¬ 
position  is  completed. 

2.  Form  the  new  aggregate  sets 

u\k  +  1)  =  |J  U,(t)  (14) 

.££<*> 

3.  Define 

£<*>  =  {t  I  £<*>)  =  ()}  (15) 

and  form  the  new  sets 

v}*+i)  =  y  v<*>  (i6) 

J,.  Construct  a  new  graph  G^k  +  l^  =  (•A^*  +  l),£(t  +  1l),  hf(k'lrl'>  =  {1, . . . ,  TV}.  For  each  pair  of  vertices 
(/,  J)  of  +  I  J ,  the  following  set  of  edges  comprise  £(*  J"1). 

(a)  For  each  edge,  e  =  (uj,  ^2,  w)  £  £(*),  where  Vj  £  E\k\  i>2  £  E{jk\  an  edge  ( I,J,w —  1)  is  added 

to  the  set  £(*  +  1). 

(b)  For  each  edge,  e  =  (t/i,t,w)  £  £(*),  where  Vi  £  t  £  T^k\  such  that  J£(M(t,  E^P)  <  oo, 

the  edge  (I,  J,  w  —  1  +  J£(k)(t,  Ejk '))  is  added  to  the  set  £(*  +  1). 

5.  The  graph  +  is  “ reduced ”  by  retaining  only  the  lowest  weight  link  between  any  pair  of  vertices 
producing  the  graph  G^*+1^  =  (hf^k+1\£^k+1^).  Specifically, 

(a)  Set  £(i  +  1)  <—  £(*  +  1). 

(b)  For  each  edge  e  =  (vi,V2,w)  £  £(*  +  1),  remove  edge  e  if  there  is  some  other  edge  e1  = 
(vi,  t»2,  w1),  w1  <  w  remaining  in  £(*  +  Il 

Set  k  «—  k  +  1-  Go  to  step  I 


□ 

Note  that  the  steps  involved  are  numbered  in  a  fashion  to  parallel  the  steps  in  Algorithm  1  with  the 
exception  that  the  last  step  is  not  performed  in  the  previous  algorithm.  The  major  computational  require¬ 
ment  of  this  algorithm  involves  determining  the  shortest  paths,  the  d  terms.  This  can  be  accomplished 
using  a  variety  of  standard  graphical  algorithms.  Since  all  shortest  paths  from  non-final  vertices  to  all 
final  classes  are  required,  an  algorithm  such  as  a  modification  of  the  Floyd  algorithm  [6]  can  be  employed. 

Algorithm  3  (Shortest  path)  Begin  with  a  graph  G  =  {/!,£),  hi  —  {1  £  =  {(*,ji,dtj)  |  »,  j  G 

hi},  and  a  set  of  vertices  E  which  may  not  be  used  as  intermediated  vertices. 

1.  Set  =  min(d,j  |  (»,  j,d,})  £  £).  D =  oo  if  there  is  no  edge  from  i  to  j. 

2.  For  n  =  0, . . . ,  N  -  1 

if  n  £  E  then  otherwise 


£>j"+1)  =  min 


3.  dE(i,j)  =  Dip 

□ 

A  simple  example  is  considered  using  Algorithm  3  in  order  to  clarify  the  approach.  The  Markov 
process  considered  appears  in  Example  1  where  Algorithm  1  is  demonstrated. 


Figure  4:  Perturbed  Markov  process  and  associated  graphs 


Figure  5:  0(l/e)  time  scale  graph 
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(a)  G<2> 


(b)  G<2> 


Figure  6:  0(l/e2)  time  scale  graphs 


Example  2  The  generator  .4l°l(e)  and  the  associated  graph  G (°)  are  shown  in  Figure  4.  The  graph  G q°^ 
has  only  the  edges  (3, 1,0)  and  (3,2,0).  The  final  classes  of  the  graph  G^  are  2?[0^  =  {1},  E ^  =  {2}, 
and  e[°^  =  {4}  and  the  non-final  set  is  T =  {3}.  Using  the  definition  in  equation  (15),  E^  =  {1,3}, 
E 2°^  =  {2,  3},  and  E^  =  {4}.  The  distances  in  the  graph  G to  the  final  classes  of  G^  are  J£( 0)(3, 1)  = 
dBi„){ 3,2)  =  0  and  «fe(<l)(3,  4)  =  1.  The  sets  UM  andV <l>  areU[l)  =  {l},!/'1'  =  {2},  and  U{31)  =  {4}, 
and  Vp1  =  {1,  3},  =  {2,  3},  and  v|15  =  {4}. 

The  graph  G^\  which  in  this  simple  example  is  already  reduced  and  therefore  also  the  graph  G^l\  is 
shown  in  Figure  5.  The  sets  constructed  in  this  iteration  on  the  algorithm  (k  =  1)  are  =  e[ ^  =  {1,  2}, 
4l)  =  =  {3},  TW  =  {},  U[7)  =  {1,2},  v[2)  =  {1,2,3},  and  W<2)  =  V<2)  =  {4}. 

Finally,  the  graph  G^  is  constructed  as  shown  in  Figure  6(a).  The  reduction  in  this  case  corresponds 
to  removing  one  of  the  duplicated  links  from  vertex  1  to  2.  The  sets  constructed  in  this  iteration  are 
=  {2},  E[ =  {1,2},  and  —  {1}.  The  set  of  graphs  and  the  sets  W(’)  and  are 
illustrated  in  Figure  7.  This  should  be  compared  to  the  set  of  Markov  processes  illustrated  in  Figure  3 
which  is  obtained  using  Algorithm  1.  □ 


4  Conclusions 

The  graphical  algorithm  presented  in  this  paper  is  significant  in  that  it  allows  construction  of  an  approx¬ 
imation  of  the  behavior  of  &  singularly  perturbed  Maikov  process  in  which  the  approximation  preserves 
the  multiple  time  scale  structure  of  the  system  while  not  attempting  to  preserve  the  detailed  dynamics 
at  any  time  scale.  Furthermore,  there  is  no  restriction  on  the  ergodic  structure  of  the  process  at  any  time 
scale.  Specifically,  there  may  be  transient  states  at  any  of  the  intermediate  time  scale  models. 

An  accurate  approximation  of  the  multiple  time  scale  structure  has  many  applications.  In  particular, 
complex  systems  which  inherently  have  transient  states  can  be  analyzed  using  the  new  algorithm.  Such 
models  arise  in  the  analysis  of  fault-tolerant  systems.  If  failure  of  the  system  is  associated  with  some 
combination  of  failures  of  critical  components,  then  complicated  multiple  time  scale  structure  may  occur. 
The  graphical  algorithm  presented  in  this  paper  provides  a  way  to  determine  the  critical  sequences  of 
events  in  the  system  which  may  lead  to  failure  and  the  order  of  magnitude  of  the  time  till  such  events 
occur. 

Another  aspect  of  this  algorithm  which  is  introduced  in  [1]  is  that  with  a  modest  extension,  the  mul¬ 
tiple  time  scale  structure  of  systems  with  multiple  perturbation  parameters  can  be  considered.  Though 
perturbed  Markov  processes  with  two  perturbation  terms  have  been  considered  [7],  that  method  requires 
making  assumptions  about  the  relative  orders  of  magnitude  of  these  parameters.  In  the  extension  of 
the  graph  algorithm,  a  set  of  multiple  time  scale  structural  decompositions  can  be  derived  where  each 
decomposition  is  associated  with  a  set  of  constraints  on  the  relative  orders  of  magnitudes  of  the  pertur¬ 
bation  terms.  These  constraints  are  associated  with  different  paths  in  the  graphs  being  associated  with 
the  shortest  paths  between  sets  of  states. 

Finally,  the  graphical  algorithm  allows  identification  of  state  transitions  rates  which  are  sufficiently 
small  so  that  they  cannot  affect  the  full  multiple  time  scale  approximation  formed  using  Algorithm  1. 
This  observation  would.allow  simplification  of  each  of  the  generators  A^^e)  by  setting  certain  transition 
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rates  to  zero  between  iterations.  The  graphical  decomposition  algorithm  could  in  this  way  be  used  as  an 

initial  analysis  to  reduce  to  computation  required  to  construct  the  complete  uniform  approximation. 
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